A Model for the Representation and Processing of Shape in Coverbal Iconic Gestures

A Computational Model for the Representation and Processing of Shape in Coverbal Iconic Gestures

When humans describe the shape of objects, they often use iconic gestures to depict what they want to convey to a listener. Gesturing gives them the ability to express spatial concepts directly in the spatial medium and thus provides an important communicative resource for spatial language. In order to harvest this resource in language comprehension systems, the composite signal conveyed in two...

Processing Iconic Gestures in a Multimodal Virtual Construction Environment

In this paper we describe how coverbal iconic gestures can be used to express shape-related references to objects in a Virtual Construction Environment. Shape information is represented using Imagistic Description Trees (IDTs), an extended semantic representation which includes relational information (as well as numerical data) about the objects’ spatial features. The IDTs are generated online ...

Coverbal Iconic Gestures for Object Descriptions in Virtual Environments: An Empirical Study

Logo and Cover Credits

In the description of object shapes, humans usually performiconic gestures that coincide with speech (are coverbal). Marked by asimilarity between the gestural sign and the described object, iconic ges-tures may easily depict content difficult to describe using words alone.Though the expressive potential of iconic gestures in human-computercommunication is generally acknowle...

Understanding Coverbal Dimensional Gestures in a Virtual Design Environment

Todays multimodal systems, which allow full-body (3D) gestures and speech as input modalities, are quite restricted to easily interpretable coverbal gestures with a prede ned shape and meaning. In this paper, we propose methods to abstract the concrete shape of gestures by using high-level features and to integrate them with coexpressive words using their phonological attributes. The applicatio...